12-29-2022, 10:03 PM
Cycle 3, stage 4 (month 2), day 29:
I have only sparsely written to my journal in this last month. Therefore, I have collected a set of ideas that I wanted to write about. This might sounds like the ideas are unconnected or out of order. I will write them down and hopefully, this will help clarify my thoughts.
I have finally overcome my procrastination concerning one specific item. I had on my todo list to create a special client specifically for measuring with accuracy the reaction time of my code. This is incredibly useful. AS soon as I make an improvement, I can validate if it does indeed make a significant difference or not. I say procrastination but maybe it was not that. I got a bunch of other tasks on my lists and I did tackle one by one in the last 2 months. It is just that this specific task never won my heart over the other candidates. The weird thing is that the more I was postponing doing it, the bigger the task did appear to be up to a point where I may even have felt stress by thinking about it. Finally, when I did it, it did turn out to be not too hard. Much less than my imagination might have made me believe.
In last entry, I was mentioning that I did enable a new io_uring feature for better performance. After a day or 2, I did realize the new code was in fact not used at all. When I did fix it and tried to run it, all hell break loose on me... Everything did stop working. I may have even felt some desperation... After some dedication, I have finally made sense of what I was observing. I made some more fixes and the new feature did finally work for real this time...
As always, when you hunt down problems, it makes you review code and by revisiting it, it makes you see new opportunities for improvements. The most notable, is that I have enabled 2 new additional io_uring features improving its performance. If I did not have to work to find the issues why multishot was not working, I would never have discovered those 2 low hanging fruit.
And finally, once it did work, I have immediately measured the code reaction time and I have been blown by the result. Enabling this single feature did make me win over 1 msec in the reaction time. This is simply amazing. Prior to that, I did spent a month and half trying to improve performance. The code did change a lot but by not focusing on the hottest bottlenecks, the huge amount of effortd did not pay off well. I may have shaved 100 uSecs or something insignificant like that...
Next, I wanted to try the combo multishot + ktls.
ktls is a new feature that I did work on last month. I had the impression that it was helping with performance but I did not measure. This was prios the creation of my new measurement tool. I did disable it because, while using it, it was causing the kernel to crash. I was suspecting maybe some new immature kernel code causing the crash. but since then, several things did happen:
Severel new versions of the kernel has been released since my last attempt
Working the code for supporting multishot made me improve the client code and several small bugs got resolved
I have learned how to create a kernel dump in case of a crash so that I can debug the crash if it occurs again.
The multishot + ktls version has been running for about 6 hours. So far so good. No crash in sight. If I remember correctly, the crash was happening after about 3 days of non-stop execution... We will see how it goes.
Again, separately, multishot and ktl are working fine but together, it did not... It would not have been fun if it did on first attempt. I guess, it is a good thing to test the code with several different configurations. By doing so, it uncovers a bunch of small glitches that would have remained hidden if I did not experiment. This resulted into a good catch! And again, I did improve some more the code which in the very hot path...
Once the combo multishot + ktls did work, I have immediately measure its performance. Unfortunately, it is not faster. It is in fact slower. Multishot alone appears to be better than multishot + ktls...
So maybe tomorrow, I'll revert back to the simpler multishot only setup... but I am keeping all the fixes and improvements that trying out the combo did allow me to make!
At last but not the least, In parallel to everything that I have written, I have seriously started the effort to build the multi exchange arbitrage idea. I have layed out the full design and I have created a roadmap to achieve this goal. I have a clear mental picture on how to reach this goal. I got those feelings in the past. I have it, I pretty much know that it is in the pocket... This will work and I will reach the goal of a working system...
I have only sparsely written to my journal in this last month. Therefore, I have collected a set of ideas that I wanted to write about. This might sounds like the ideas are unconnected or out of order. I will write them down and hopefully, this will help clarify my thoughts.
I have finally overcome my procrastination concerning one specific item. I had on my todo list to create a special client specifically for measuring with accuracy the reaction time of my code. This is incredibly useful. AS soon as I make an improvement, I can validate if it does indeed make a significant difference or not. I say procrastination but maybe it was not that. I got a bunch of other tasks on my lists and I did tackle one by one in the last 2 months. It is just that this specific task never won my heart over the other candidates. The weird thing is that the more I was postponing doing it, the bigger the task did appear to be up to a point where I may even have felt stress by thinking about it. Finally, when I did it, it did turn out to be not too hard. Much less than my imagination might have made me believe.
In last entry, I was mentioning that I did enable a new io_uring feature for better performance. After a day or 2, I did realize the new code was in fact not used at all. When I did fix it and tried to run it, all hell break loose on me... Everything did stop working. I may have even felt some desperation... After some dedication, I have finally made sense of what I was observing. I made some more fixes and the new feature did finally work for real this time...
As always, when you hunt down problems, it makes you review code and by revisiting it, it makes you see new opportunities for improvements. The most notable, is that I have enabled 2 new additional io_uring features improving its performance. If I did not have to work to find the issues why multishot was not working, I would never have discovered those 2 low hanging fruit.
And finally, once it did work, I have immediately measured the code reaction time and I have been blown by the result. Enabling this single feature did make me win over 1 msec in the reaction time. This is simply amazing. Prior to that, I did spent a month and half trying to improve performance. The code did change a lot but by not focusing on the hottest bottlenecks, the huge amount of effortd did not pay off well. I may have shaved 100 uSecs or something insignificant like that...
Next, I wanted to try the combo multishot + ktls.
ktls is a new feature that I did work on last month. I had the impression that it was helping with performance but I did not measure. This was prios the creation of my new measurement tool. I did disable it because, while using it, it was causing the kernel to crash. I was suspecting maybe some new immature kernel code causing the crash. but since then, several things did happen:
Severel new versions of the kernel has been released since my last attempt
Working the code for supporting multishot made me improve the client code and several small bugs got resolved
I have learned how to create a kernel dump in case of a crash so that I can debug the crash if it occurs again.
The multishot + ktls version has been running for about 6 hours. So far so good. No crash in sight. If I remember correctly, the crash was happening after about 3 days of non-stop execution... We will see how it goes.
Again, separately, multishot and ktl are working fine but together, it did not... It would not have been fun if it did on first attempt. I guess, it is a good thing to test the code with several different configurations. By doing so, it uncovers a bunch of small glitches that would have remained hidden if I did not experiment. This resulted into a good catch! And again, I did improve some more the code which in the very hot path...
Once the combo multishot + ktls did work, I have immediately measure its performance. Unfortunately, it is not faster. It is in fact slower. Multishot alone appears to be better than multishot + ktls...
So maybe tomorrow, I'll revert back to the simpler multishot only setup... but I am keeping all the fixes and improvements that trying out the combo did allow me to make!
At last but not the least, In parallel to everything that I have written, I have seriously started the effort to build the multi exchange arbitrage idea. I have layed out the full design and I have created a roadmap to achieve this goal. I have a clear mental picture on how to reach this goal. I got those feelings in the past. I have it, I pretty much know that it is in the pocket... This will work and I will reach the goal of a working system...
