+- Subliminal Talk (https://subliminal-talk.com)
+-- Forum: Men's Journals (18+ NSFW) (https://subliminal-talk.com/Forum-Men-s-Journals-18-NSFW)
+--- Forum: Men's Journals (https://subliminal-talk.com/Forum-Men-s-Journals)
+--- Thread: lano1106 UMSv2 (/Thread-lano1106-UMSv2)
RE: lano1106 UMSv2 - lano1106 - 11-27-2022
Cycle 3, stage 4, day 27:
I have experienced another strike of luck today... An odd system behavior did manifest itself in front of me this morning... It took me few hours to nail it down but this was a big ugly regression bug that I have inserted during my performance refactoring that must coincide with the start of stage 4.
I have experienced very bad trading performance since then and I have even complained about it here (day 22 entry) and the bug that I have finally discovered today is certainly the culprit of this bad luck...
I have really been present at the right place and at the right moment. Without the good luck that I had today, the 1 liner bug could have stayed unnoticed for much more longer and cause trouble to my system profitability!
RE: lano1106 UMSv2 - lano1106 - 12-04-2022
Cycle 3, stage 4 (month 2), day 4:
I have been annoyed today because I had a very simple plan about things that I wanted to achieve and I got interrupted the whole day by new issues.
OTOH, the issues that did pop out are very good catches so I should be grateful for them to cross my path...
RE: lano1106 UMSv2 - lano1106 - 12-12-2022
Cycle 3, stage 4 (month 2), day 12:
Markets are very quiet since FTX debacle... idk for sure if it is part of the aftermath or if it is some sort of seasonality... or perhaps both...
Anyhow, since the last journal entry, I have completed the latest chapter of my efforts to finally break into algo trading success and I need to conclude that I need one more time to digest a setback.
Once again, I did follow my intuition despite having a small voice telling me that what I was doing was not the priority...
I have acquired a new groundbreaking skill to optimize my code. Despite having the impression that it was already pretty robust and mature prior the start of this latest attempt to improve things, with this new skill, I must have found ways to improve the program performance by a good 20%... it seems like a lot but if it does everything in 100uSecs but I would need to be faster by 10 mSecs, it is simply a sword swing into water. It did nothing...
By its real-time nature, it was already doing things fast enough within a certain time-frame to stay in the real-time realm but by being faster, it is getting closer to immediacy. Working on this was extremely stimulating and rewarding... However, now that I am done, I have no other choice to admit that in terms of end-result it did change nothing.
Hopefully, I have learned something out of that experience... Hopefully more wisdom to not pursue self-deceiving good ideas... The whole effort must have lasted a good month... more or less...
To not be too hard on myself, I must acknowledge cognitive bias... When I just complete a superb improvement and I launch it, no matter how not perceivable the difference is, I always have the the impression that the logs scroll much faster in the terminal... If my system experience a very good trading session right after an upgrade, I will of course conclude to a causal effect with the latest addition... Usually, however, it is simply random and coincidences...
Being in a bad trading period might have nothing to do with my work in the last month. The poor trading and the effort put into improving the system are just unrelated... Cognitive bias makes my mind create a link where there is maybe none...
If I want to reframe the whole thing positively, it must be that now, I have an unshakable belief that my issue is elsewhere than in my system code...
RE: lano1106 UMSv2 - lano1106 - 12-20-2022
Cycle 3, stage 4 (month 2), day 20:
I have discovered a small bug very subtle hard to detect but that was defeating any chance of success of my market making strategy that I must have introduced something like 2 weeks ago by doing a minor aesthetic only change.
this is the second time that this happens. The first time that I did report such incident was in the journal entry for Cycle 3, stage 4, day 27.
It is a bit weird that the same error occurs 2 months in a row during stage 4 runs...
A new feature that I did add to my system is to create a daily report of the most traded cryptos. This was a free report that was sent out by email by the exchange but my exchange stopped producing the report since the beginning of December presumably the person in charge of the report has been lay off in the last 30% staff cut annonced last month...
I woke up on Sunday morning and felt a urge to have this info at hand. It was not very hard to do as my system already collect all the info needed to build this daily report. It was fun to do and I got rewarded by an amazing insight that was not included at all in the exchange report.
I thougth that the most traded asset was BTC. It is not!
USDT has more than twice more volume than BTC...
RE: lano1106 UMSv2 - lano1106 - 12-21-2022
Cycle 3, stage 4 (month 2), day 21:
Today went fast. I have completed an interesting feature by integrating the latest io_uring recvmsg multishot feature into my client. It has been introduced with the Linux kernel version 6 so it is still hot out of the baking oven...
io_uring is a very special beast. It is an API that is not meant for the faint of heart. It is possibly in my top 5 of the most complex API that I have ever interacted with and they keep adding more and more complexity all the time. I'm quite happy to have on boarded the io_uring boat long time ago. It allows me to digest incrementally each addition to it... It could have been a bit overwhelming to have to integrate all of it in a single shot...
Yet, despite its complexity, it must also be the most efficient way to perform i/o on Linux...
Ok, this will be enough infrastructure code work for this week. Tomorrow, it will be application code... I need to develop the code needed to make arbitrage between different exchanges. For doing that, a prerequisite will be to perform algorithmically transfers. This is something that I have never touched since the start of my journey.
I expect this feature to be quite trivial...
RE: lano1106 UMSv2 - lano1106 - 12-29-2022
Cycle 3, stage 4 (month 2), day 29:
I have only sparsely written to my journal in this last month. Therefore, I have collected a set of ideas that I wanted to write about. This might sounds like the ideas are unconnected or out of order. I will write them down and hopefully, this will help clarify my thoughts.
I have finally overcome my procrastination concerning one specific item. I had on my todo list to create a special client specifically for measuring with accuracy the reaction time of my code. This is incredibly useful. AS soon as I make an improvement, I can validate if it does indeed make a significant difference or not. I say procrastination but maybe it was not that. I got a bunch of other tasks on my lists and I did tackle one by one in the last 2 months. It is just that this specific task never won my heart over the other candidates. The weird thing is that the more I was postponing doing it, the bigger the task did appear to be up to a point where I may even have felt stress by thinking about it. Finally, when I did it, it did turn out to be not too hard. Much less than my imagination might have made me believe.
In last entry, I was mentioning that I did enable a new io_uring feature for better performance. After a day or 2, I did realize the new code was in fact not used at all. When I did fix it and tried to run it, all hell break loose on me... Everything did stop working. I may have even felt some desperation... After some dedication, I have finally made sense of what I was observing. I made some more fixes and the new feature did finally work for real this time...
As always, when you hunt down problems, it makes you review code and by revisiting it, it makes you see new opportunities for improvements. The most notable, is that I have enabled 2 new additional io_uring features improving its performance. If I did not have to work to find the issues why multishot was not working, I would never have discovered those 2 low hanging fruit.
And finally, once it did work, I have immediately measured the code reaction time and I have been blown by the result. Enabling this single feature did make me win over 1 msec in the reaction time. This is simply amazing. Prior to that, I did spent a month and half trying to improve performance. The code did change a lot but by not focusing on the hottest bottlenecks, the huge amount of effortd did not pay off well. I may have shaved 100 uSecs or something insignificant like that...
Next, I wanted to try the combo multishot + ktls.
ktls is a new feature that I did work on last month. I had the impression that it was helping with performance but I did not measure. This was prios the creation of my new measurement tool. I did disable it because, while using it, it was causing the kernel to crash. I was suspecting maybe some new immature kernel code causing the crash. but since then, several things did happen:
Severel new versions of the kernel has been released since my last attempt
Working the code for supporting multishot made me improve the client code and several small bugs got resolved
I have learned how to create a kernel dump in case of a crash so that I can debug the crash if it occurs again.
The multishot + ktls version has been running for about 6 hours. So far so good. No crash in sight. If I remember correctly, the crash was happening after about 3 days of non-stop execution... We will see how it goes.
Again, separately, multishot and ktl are working fine but together, it did not... It would not have been fun if it did on first attempt. I guess, it is a good thing to test the code with several different configurations. By doing so, it uncovers a bunch of small glitches that would have remained hidden if I did not experiment. This resulted into a good catch! And again, I did improve some more the code which in the very hot path...
Once the combo multishot + ktls did work, I have immediately measure its performance. Unfortunately, it is not faster. It is in fact slower. Multishot alone appears to be better than multishot + ktls...
So maybe tomorrow, I'll revert back to the simpler multishot only setup... but I am keeping all the fixes and improvements that trying out the combo did allow me to make!
At last but not the least, In parallel to everything that I have written, I have seriously started the effort to build the multi exchange arbitrage idea. I have layed out the full design and I have created a roadmap to achieve this goal. I have a clear mental picture on how to reach this goal. I got those feelings in the past. I have it, I pretty much know that it is in the pocket... This will work and I will reach the goal of a working system...
RE: lano1106 UMSv2 - lano1106 - 01-09-2023
Cycle 4, stage 1, day 9:
It has been 2 days since I have switched to cellphone playback as recommended by Shannon...
and it seems like it is doing something.
for one, I have made crazy dreams for the last 2 days while my dreaming was very quiet before that. Interestingly, they have nothing to do with money. They are related to sex/love. I am walking around and seduce and date a bunch of attractive women as if I was some sort of Don Juan...
Also, some emotions have resurfaced from a traumatic event that is almost 2 years old. Some facts of the event were cloaked. Invisible to my conscious mind. I was blind to this possible interpretation of the event because the implications of that option are real ugly but now I am seeing it... I guess seeing it is a mandatory step in order to heal and move on... Maybe it is something has stopped me from succeeding since then.
Finally, yesterday crypto markets were significantly up for the first time since a long time and my system made profits on most of its trades. This has been a very long time since it did happen...
Is it coincidence or an effect of changing my playback method?
RE: lano1106 UMSv2 - lano1106 - 01-18-2023
Cycle 4, stage 1, day 18:
I am super excited to report that by following Shannon minor tweak suggestion of using a cell phone for playback instead of a PC setup appears to work very well.
Cycle 4 seems like it is going to be a totally different beast. Things appears to have unblock and it seems like I am on the verge of experiencing a major breakthrough...
RE: lano1106 UMSv2 - lano1106 - 01-20-2023
Cycle 4, stage 1, day 20:
I have been teased by success one more time. I thought that I was on the verge of reaching a major milestone. Initial results were promising.
I got 3 partially successful trades using the new experimental strategy. Of course, it did stumble into few expected snags when new code is run. In front of the snags, I did masterfully adapt the code so the system behave well in front of those new situations.
I did put in place the improved and corrected system online at the end of the afternoon yesterday and since then, the new strategy has not been triggered a single time...
Not only that but instead, an unexpected error did trigger the execution of a bunch of new code not 100% well tested and it did not react as expected.
I am kinda annoyed because instead of returning on working on another feature to create a new income generation, I had to spend few hours this morning to analyze the logs to figure out what went wrong and modify the code to correct it and/or provide better logging the new time the situation occur...
It is not that bad... This incident has the merit of having uncovered problems and the system is now improved but I am feeling annoyed because things aren't going as smoothly as I hoped.
RE: lano1106 UMSv2 - lano1106 - 01-23-2023
Cycle 4, stage 1, day 23:
My latest strategy started working over the week-end. I had to take care of few new bugs that keep coming but after having taking care of them, the strategy is working super well.
It generates an instantaneous 3-4% profit per trade. I am still in the process of tweaking the strategy parameters. ie: the less greedy the profit target is, the more frequent trades will be triggered.
I haven't found yet the optimal parameters. Maybe aiming for 2.5% profit will generate twice as much as the current 3.5% target... I am still figuring out those details... but this is definitely an enjoyable problem to work on. I am definitely in a more positive head space than I was few weeks ago... It feels good and I also feel that I deserve it with all the difficulties that I have encountered...
and this is on top of the ongoing process of supporting my second exchange... I may have this project in my head for at least a year but it is now getting seriously close to become reality. Now that I have a working secret sauce... Having the sauce running on a second platform means more or less doubling the generated income... Plus the new opportunities this is going to open up.... Plus prudently scaling up my daily trading budget now that my confidence in the system increases...
I start to see the light and it feels good...
RE: lano1106 UMSv2 - lano1106 - 01-27-2023
Cycle 4, stage 1, day 27:
It seems like I have experienced the beginner's luck with my new strategy. Despite having spent the whole week to review its executions and refine its implementation based on what went wrong, the number of generated trades did not stop decreasing.
I suspect that in the last week, market conditions did change sufficiently to explain my observation.
For the most part, this strategy did exhibit a 100% profit success rate. This was until this morning where a single trade lost 5%. I did review how it did happen but this has made me work the full day on a fix to avoid seeing this situation repeat itself again.
This situation made me think about a very important market aspect that I was completely ignoring until now. I had some sort of vision or a revelation of a plan into how to integrate this aspect in my system.
I am not sure how long it will take to realize this new vision. I estimate that maybe a day will be enough. So tomorrow, I could maybe have a working prototype to try out. However, I am notoriously bad to make accurate estimates. So worse case scenario, it is going to take me the full week-end... but I am optimist that I am able to complete the task in one day...
It is so clear in my mind... but when it comes to write down... It is always more complex and difficult in reality...
RE: lano1106 UMSv2 - lano1106 - 01-29-2023
Cycle 4, stage 1, day 29:
I am in the process of realizing my latest vision. I got interrupted by a new problem popping up. The issue has always been present but the right conditions to have the problem did never show up before today. It is always a bad timing for these bugs to appear but I force myself to see them as a life gift, an act of luck.
RE: lano1106 UMSv2 - lano1106 - 01-31-2023
Cycle 4, stage 1, day 31:
I am going to start stage 2 tomorrow. It is time to summarize how stage 1 went.
It was fantastic. Today, I have completed a major feature. This is the right timing to have a nice stage closure...
That being said, I have not been able to nail down good strategy parameters for my latest strategy creation. It experience sporadic success but it is not enough to generate a sustainable income... I'll keep continue to fine tune those parameters while at the same time continue porting my system to a second exchange like I have dreamed of for at least a year...
I think that I am procrastinating this task big time...
RE: lano1106 UMSv2 - lano1106 - 02-01-2023
Cycle 4, stage 2, day 1:
I like how stage 2 is starting. I have a constant flow of good ideas and good solutions coming into my mind...
If this remains like that, I feel like success is pretty much inevitable...