cycle 2. Stage 3, day 17:
I did stumble into another spectacular improvement. Many months ago, I took great care to eliminate every memory allocation out of the hot path.
The most insidious allocation is the one that you don't even suspect that it is there at all...
I attribute again this powerful discovery to luck.. Yes one more time, luck... I say that very often since I have started to use UMS.
Here is the chain of thoughts that made me discover this gem.
After dinner, I was reading one of my favorite news website and I have found an article announcing the new release of a memory allocator lib:
https://www.phoronix.com/scan.php?page=n...3-Released
This made me think that I have never really did run a heap profiler on my software. I did attempt to run valgrind on it but for some reason, my app is corrupting valgrind and nothing good can come out of a valgrind run. I suspect that it might be because my app is launching kernel threads through io_uring and valgrind does not like that at all...
Anyway, I did search the net to see if I could find another good heap profiler that will run nicely with my app. I did find heaptrack:
https://github.com/KDE/heaptrack
and it works very well...
This made me discover that my #1 memory allocation spot was inside std:: stable_sort(). I even had no idea that this function was allocating memory. This function is called something like 4 million times every 5 minutes or so. I have replaced calls to stable_sort to something else not allocating. I am currently testing this new improvement... Not sure yet by how much it will improve my reaction time but it should make a difference...
cycle 2. Stage 3, day 18:
Since yesterday or since the beginning of the week, I have started to study the new language features from C++17 and above and as I learn about it, I modernize my codebase.
I have discovered a new language addition in an area that is in the hot path of my system and deploying this new language feature should make a nice performance improvement.
Also, going through the whole code base is a new opportunity to review old code and it turns out that it did allow me, out of pure luck again, to discover 2 small glitches in one of the core classes that is also in the hot path.
I am really eager to deploy a new version with those improvements in. The modernization effort is tedious and huge. I have been doing only that the whole day today but I have hope to complete sometime tomorrow.
cycle 2. Stage 3, day 19:
We are early in the morning so I'll probably have more to say later but the first time I look at my system it made me see a 10% loss.
this loss is painful considering that my average daily profit is around 3%.
The loss did occur with 2 regular arbitrage trades. Those type of trades contrast with my newer micro trading strategy where the size is about $3. If something goes wrong with micro-trading, I might not even notice because the amounts are so small...
After about 30 minutes of postmortem analysis, it seems like the situation is not as bad as I originally thought. I have already identified 2 big problems that should not be too difficult to fix and I have found funds stuck in some intermediary asset balance making my loss more like 4% instead of 10%...
Those errors are painful but needed to find problems and fix them...
cycle 2. Stage 3, day 24:
I have fixed the problems that I have mentionned in the last entry. It took me few days to do it. While I was fixing it, the system containing the problem kept running. This made me nervous a little bit that the issue reoccur. It did not. It was really a very rare occurrence problem but a small part of the fix could have a negative impact on the execution of many trades without making them negative. So I might see an improvement with this fix.
My code base modernization is completed. I did insert 2 bugs while doing it. Really stupid errors. 1 has been relatively easy to find. It took me 15 minutes. The other one took me something like 2 days to figure out. This hunt gave me 2 benefits. I have discovered 2 new tools:
cppcheck
fsanitize address
with the help of those tools. I have uncovered one really bad and unrelated memory corruption issue that was silent so far and I have finally nailed the hunted bug.
The modernization that was coming along with few performance improvements did appear to have worked. I did measure a significant improvement in my system reaction time. It has finally broke the 30ms value few times. The minimal theoretical minimum must be in the 15-20ms range.
One trick that I have discovered in the last few days, it is to watch in the background presentations from C++ conferences on YT while I work. This has been a source of inspiration for new improvement ideas. So far, this practice has been very valuable...
I have noticed something very important. My monthly trading volume and my income is directly proportional to the value of the traded asset. In the last 2 weeks, my income nominated in crypto has remain stable. Nominated in fiat, it has melted roughly in half since the crash. I see this in my daily profit number and by observing my trading volume slowly deflate. It has peaked at 1.1M maybe a month ago to be about go under 600K in the next few days...
I am not sure exactly what I can do to fight this. I need to improve my system to make it more efficient. If I succeed, this will translate to double my income once the crypto price returns back to their previous level... Anyway, I use that phenomenon as a source of motivation. At the current pace, if the trend continues, my trading volume could go under 500K and this is something that I do not want to see happen. It is like a train wreck in slow motion... I must have about 2 or 3 weeks before it happens...
Finally, I am eager to move on to stage 4... It is not obvious what stage 3 did contribute this time to my journey. At least, if it did something, it has been very stealthy... under the surface type of changes... It contrast a lot with my major breakthrough experienced during stage 2...
Next 2 months of stage 4 are planned... If I am lucky the end of the UMS cycle 2 will coincide with the release of DMSI 5.9g... I believe that after 5 months of substained effort and focus to achieve UMS... A small treat with a short DMSI run is something that will be well deserved...
Update:
Journaling is really beneficial. I have just reread this post and it made me realize something. Something that I was supposed to do that I did not do in the last month.
Support new exchanges. By assuming that my system will be profitable out of the box on any exchange, I can easily and predictably increase the generated income by a factor equal to the number of added exchange support. If I add 1, this could double my income. If I add 2, it will triple it and so on...
It seems like occupying myself with many small less important tasks is my unconscious strategy to procrastinate on doing something that is really game changing that I fear doing...
it is not to say that everything that I did during the last month was worthless... far from it... The quality of my system did significantly improve and this will be beneficial for life from now on... It just to mean that I got a spark of lucidity... some sort revelation of what needs to be done next...
cycle 2. Stage 3, day 27:
Despite what I wrote in my last entry instead of doing right away what I said that I would do, I did work on something else. I had several reasons for doing so.
First, 2 months ago when I got my last breakthrough with the fee free trading, I was in the middle of implementing an idea. This task has been put on hold since then. I hate placing an idea on hold for so long because all the small details that were clear in my mind have the time to disappear and it is extremely inefficient to return back to old tasks and rediscover all the details that were available in the past. The longer you wait, the costlier it is.
Second, I am competitive in nature. Despite not representing a big chunk of my trading volume, it piss me off to miss a trade because my system has been too slow. Despite all the optimizations that I did bring to it in the last month, I was witnessing juicy trade misses in my log window as I was in front of my computer.
The idea that has been put on hold is that currently when my system receive a price update. It puts the new price in place and then reevaluate all the trade combinations that may be affected by the change and if it finds a profitable combination, it launch a trade. All this computation is kinda complex and expensive. I got the idea of batch updating. That is instead of reevaluating everything at every update, I should empty the TCP reception buffers and place all the updates it contains and only when all the updates have been put into place, perform the reevaluation. This small change had the potential to spare thousands of duplicate reevaluations to my system hence make it more reactive and increase its processing capacity.
The concept is simple but the implementation did require some design modification. I rarely do that but this time, I had to sit down and write some design document to figure out how I would do it. Usually, I just have a mental picture of what I want to do and I go straight to coding but I felt that this was not an option this time. It must have taken a day or 2 to just create the design.
and all in all, I spent the whole week on this effort including coding. It appears that everything went fast but it is me that was so absorbed by the task that I did not see the time pass. (we are already Friday!)...
Now the expected results. I was expecting that I would make my system faster and it would finally succeed grabbing the few trades that it currently miss. Yes, I got that but I got way much more than that. Before this change, fee free trading was representing 95% of all my trading. It steadily profit every day but due to its nature, it was literally generating only pennies of profit. Since yesterday, regular arbitrage trading has become the predominant type of trade. It is beautiful to see the system operating because the last time where it was doing a decent amount of arbitrage, my system did immensely gain in maturity and robustness with all the bug fixes that I did put into it in the last several months... and regular arbitrage trades are much juicier than fee free trading. We are talking 2,3,4% and sometimes even more on amounts of few hundred dollars.
So yesterday my monthly trading volume fell below 600K and was getting dangerously close to the value of 500K that I absolutely did not want to see my volume go below. I did wake up this morning and the volume downtrend has finally broke up! My volume must be about 650K and in a single evening, I must have generated as much profit than I did in the last week!
I am not too sure why I got this result. My best explanation is that the exchange was throttling down the pace it was sending me updates to adapt to my system processing capacity and several updates were just dropped making several trading opportunities simply invisible to me.
In conclusion, it seems like stage 3 theme was system performance improvement. 2 major breakthrough in 2 consecutive months! It is also a nice example of not doing what your conscious mind tells you to do and you go against reason and stick to what your guts is telling you... It would have been so easy to pass beside this hidden opportunity and not even suspect that it was there... I am so grateful for everything that is happening. My multi year effort finally starts to pay off... For the records, I started this effort in December 2019...
Once again, it seems like I am going to confirm the 3 years rule... This is the amount of time that it usually takes to bring a new venture from scratch to profit...
Update:
I got few glitches during the last run. I never stop to amazed by how many random ways that stuff can go wrong... The luck to stumble into all those glitches is awesome
cycle 2. Stage 3, day 28:
I have been misled into thinking that my batch update feature was a game changer feature. It turns out that its introduction did coincide into a burst of market volatility making the trading activity significantly increase.
The dust has settle down and the system activity has returned back to a more normal level...
This is not the first time that this type of false victory happens.... Luck is a mysterious beast... Coincidences are unbelievable...
oh well... the batch update certainly does not hurt performance... I still have plenty of improvement ideas in my list...
One is currently taking shape in my mind...
cycle 2. Stage 3, day 30:
I have refactored my code in a way that let me profile the software without sending out orders by creating a mock-up execution engine.
Doing so made me realize that some code could be factored out and put into a new class to ease its reuse.
Each time that I rework the code, this makes me see opportunities for small improvements. This code that is in the hot path has been improved in at least 3 different ways... Nothing groundbreaking but small improvements do have a cumulative effect in the long run...
Today markets were mostly green. It seems like it is something that my system react positively. I woke up with my trading volume at around 580K but the day has been very busy and the volume is about to go back over 600K.
I am going to start stage 4 tonight.
I did a task that is super tedious and boring to do. It is to add new supported pairs into my system.
There was over 20 new pairs introduced in the last month.
I am not too sure that adding them will pay back anything. About half of them are so low volume (less than 5K daily) and all of them are only tradable from USD/EUR which is not very useful for triangular arbitrage but who knows maybe hidden among them, there is gem waiting to be discovered... LUNA2 despite not being a quality asset has a very high volume from which my system can find something interesting...
I was supposed to stop optimizing my reaction time and move to other tasks such as adding new features but there is a small voice telling me that I can still do better... I took a half hour break to relax and there is indeed few excellent ways into which I can make my system faster that did pop out into my mind...
The current performance is still not satisfactory... It swings from being the best that iI have never seen to way too slow and having my ass handed to me by my competitors... I could move on but again the small voice in my head tells me to persist in the optimization effort.
Today's summary report shows that despite having made a nice and better than usual profit, the value of the missed trades is close to the double of the profit amount...
cycle 2. Stage 4, day 1:
Today, I spent the whole day elaborating the design of the idea that I did mention in the previous journal entry.
I have layed out a pretty detailed design but before proceeding to implement it, I have the bright idea to instrumentalize this part of the code to validate that it is a bottleneck and the idea is worthwhile to pursue.
The result is that there is practically no gain to complexify this part of the system, no significant gain can be achieved there... I kinda feel dumb... I should have done my timing measurement *before* spending the whole day to create a full design...
On the positive side, I may implement a very small subset of the idea and if I am lucky, this may shave a uSec or 2 from the system reaction time but absolutely nothing that is groundbreaking...
To be fair, it has been beneficial to have a magnifying len over the #1 hot spot of my system... I have spotted few very small inefficiencies that I will be able to remove and make this module faster than it was yesterday...
cycle 2. Stage 4, day 3:
My monthly trade volume is still slowly decreasing... It used to be hoovering around 600K but it is now around 560K. and obviously nothing that I have done so far to change the situation has worked or made any significant difference in the situation.
I have prepared a new design for the scaled down feature from the idea that did not go anywhere yesterday. This is progressing well. It might be completed tonight or sometime tomorrow.
I have stumble into another memory allocation in the hot path that went unnoticed until now. The effectiveness of the system keep improving...
I love STL compliant containers... By defining a custom type with a single type aliasing statement, you can switch a container used everywhere by changing a single line and everything recompiles smoothly with a new container possessing totally different characteristics from the previous one!
This afternoon, maybe luck has struck again. I did witness something that I did not know being possible. It is reminescent to the observation that I did before the major Stage 2 breakthrough. I will validate if this goes somewhere but it may be possible to increase the system throughput by simply changing 1 line in the system operation settings... If this turns out to be correct, this will be again another manifestation of amazing luck...
oh and SuperRare (RARE) altcoin sucks! My system got sucked into a trade involving this crypto and it is currently stuck in it because of its very low volume. idk what is happening but it seems like exchanges are flooded by a bunch of shitcoins... Around 25 new coins have been introduced in the last month and AFAIK there is nothing in this bunch of coins that I would want to hold long term...
cycle 2. Stage 4, day 4:
I have stumbled into the part that has the potential to generate more profit. I guess that I was postponing this part as much as possible by taking care of different things except this one. It seems like an unconscious modus operandi to avoid tasks that I somehow associate with pain.
It is as if, stage 4 has been able to put me back on track on the things that can actually make a difference.
I am not even sure how this did happen. I was following my work plan and I sidetracked it myself. This must be the catalyst:
(06-03-2022, 04:17 PM)lano1106 Wrote: [ -> ]This afternoon, maybe luck has struck again. I did witness something that I did not know being possible. It is reminescent to the observation that I did before the major Stage 2 breakthrough. I will validate if this goes somewhere but it may be possible to increase the system throughput by simply changing 1 line in the system operation settings... If this turns out to be correct, this will be again another manifestation of amazing luck...
I have already made a small change and this is already bringing pennies that I was leaving on the table before...
cycle 2. Stage 4, day 5:
Today, I have worked under pressure. I wanted to complete a very important change and put it in operation before the time where I had to leave home to participate in an event that I have committed to come.
I did find the experiment of working under pressure interesting. I feel that this made me work faster. I wonder if it would not be possible to setup artificial deadline to trick my mind to work faster...
Bottomline, I did not succeed completing my work before leaving. It turns out that it was a good thing as it was containing defects. Having rushed to upgrade, it would have run unattended and this could have caused losses.
Yesterday evening, I was planning to complete what I left when I return but this did not work. I was exhausted and I fell asleep without being able to continue working.
One interesting event, is that I did suffer of a little bit of insomnia. A small voice kept telling me that there was something not right in the work done during the day... The intuition was indeed correct. Was that my unconscious mind collaborating to the effort? I like to think that it is the case.
This morning I have completed the task that I left unfinished yesterday but glitches have been observed in the current run. It took me the day to analyze logs to figure out what was happening. The timing sucks for that... I am so eager to launch the latest and improved version... but it did pay out to be patient. I have found what was wrong. Minor glitches in my latest pennies grabber optimization. The pennies grabber idea was great but the initial release contained glitches... It is now fixed...
So eager to try out the new version... I think that I am almost ready now...
cycle 2. Stage 4, day 12:
Stage 4 is definitely different from stage 3. My mindset has changed in the last 10 days. I have become pragmatic while I was more idealist. It seems like I am looking to work only on things that will have a direct and immediate impact on my income.
For instance, my fee free trading system was only working when markets were trending up. I have generalized the algorithm by mirroring what it was doing so that it works also for down trending markets...
It took a week and it started working yesterday. In theory, this new feature has doubled my profit but it feels like it did more than doubling...
I have another doubling income idea in the pipe... Doubling again means quadrupling the income since stage 4 has started...
cycle 2. Stage 4, day 13:
I have a tough wake-up... I could not enjoy the week-end victory for a very long time...
I have finally succeeded to reach a comfortable income to see it shrink 25% overnight with the ongoing crypto carnage after enjoying the fruit of my efforts for a mere day...
I have no other choice to double down the effort I put... One thing that I know is that once the crypto retrieve their previous high... This will make me insanely wealthy but right now it is very uncomfortable...