I've run into an issue when using Tixati 2.56 (64-bit) on Windows 10 Creator edition.

Modified Memory climbs slowly until it consumes all available memory. Even exiting or terminating Tixati doesn't release the memory, nor can it be force-release through memory management tools like RAMMap. The only way to release it is through a reboot. Even if you stop Tixati after it's only eaten a few GB, nothing releases it. I've stopped every single process, service and driver that I can without forcing a reboot.

The problem doesn't occur with any other torrent client.
It doesn't seem to occur in Tixati with any number of small (<50GB files)
The memory usage climbs in proportion to the amount of traffic. (no, it's not a Killer NIC. It's a RealTek GbE and I've even used an alternate NIC and driver. No luck.)
It's not limited to a specific torrent. It seems to be correlated to the size of the file being downloaded. A single 100GB+ file creates the issue, but 50 x 2GB pieces in a single torrent doesn't seem to.

I managed to track down what was happening.

One particular drive was throwing an I/O error which was then preventing any subsequent writes to the drive. The error didn't show up in the event log and wasn't trapped by the application. It only showed up when I started poking around in threads to see what was going on. Once the error showed, nothing reset it and the Modified list just grew with every byte that Tixati received. Any attempted writes to the drive just went to the modified list and never committed.

I should have clued in that I'd started using some space on a different volume around the time the problem started, but I missed the connection.

The odd thing is that the problem only showed up in Tixati. When I rebooted and connected a different client to the incomplete torrents, they completed just fine.
I started Tixati up with some new torrents, waited for the problem to occur and then turned off write-caching on just that volume and now the problem is gone completely.  

It's got to be some weird incompatibility with write-caching on that specific volume. It works fine on all of the other drives and volumes on my system.

I'll have to run Spinrite against that drive and see if something's up. But I suspect that it's not an actual drive issue since things are working great with the cache disabled.

In any case, it's not a Tixati bug. Tixati's just surfacing a problem that other software doesn't.

In case somebody else runs across this same problem or Tixati devs read this, I'm letting you know that this problem still persists as of July 2022. Just started downloading and seeding hundreds of 50-100 GB torrent files onto an external HDD and it stops downloading after a few minutes as the memory fills up to my PC's maximum capacity. Any attempt to touch the Tixati UI like right clicking will freeze the app completely. Closing and relaucnhing or changing settings or relaucnhing the target drives does nothing, only restarting the PC can fix the issue albeit temporarily before the issue arises again. Been with Tixati for a decade, I am sad to let it go today. Hope it gets fixed in the future.

Try lowering Memory priority for tixati, for utorrent i used this and helped me:

reg add "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\Tixati.exe\PerfOptions" /v PagePriority /t REG_DWORD /d 1

save as .bat then run as admin.