cycle 2. Stage 4, day 15:
Maybe sometime last week-end, I did introduced a nasty problem that was making the server crash once per day. It took me some time to figure out the cause. I think that I have finally nailed it sometime this morning.
The new version is still operating with great performance. In retrospect, last week-end on the first 2 days of operation, the performance was just insane while on the surface the markets were quiet. It seems like the system has been able to surf on an invisible wave created by some powerful big players that had insider info to what was to come early this week.
Finally, I have noticed something about my own behavior. Few days ago, I was planning to work on the next feature that could again double the system income. I did put the feet a little bit on the break pedal for this project.
idk if it is fear or prudent precaution but I just feel currently overwhelmed by the volume of activity generated by the system. Many aspects of the system that were adequate for reporting its activity before the last update have now become inadequate due to the increased volume.
I need to adapt and stabilize the system for its new reality before continuing going forward. Plus, I have a suspicion that maybe few small bugs remains lurking that may negatively impact the overall system performance. I feel the need to observe the system and convince myself that all is good.
Fixing bugs and improving a system robustness is much easier when the volume of activity is under control. I feel very important hunt for bugs before increasing again the amount of activity. Maybe I need to adapt to the new reality too. I feel like UMS goal is getting close to be concrete in significant ways.
cycle 2. Stage 4, day 16:
During my shower this morning, I came up with an ambitious plan to finally go for the bull eye task. My need for money is becoming more than serious....
I did figure that if I could come up with the feature completed in order to MULTIPLY the system income, it was worth giving it a shot.
My plan got derailed. My system did exhibit 2 stuck trades for unexplained reasons. Investigating why has just drained the full day.
Even now, it is not 100% conclusive. I fixed a small bug... I did tweak some execution parameters but the issue is beyond my control... There is a limit about the floating points variables and sometimes they are going to introduce some weird round errors.
I have reproduced the problem to its simplest form:
double price = 0.00499999;
trunc(price*100000000.0)/100000000.0
= 0.00499998!
One issue remains a mystery. On the bright side, this whole experience did allow me to develop new testing methods...
I have instrumentalized further the code so that if the issue reoccur, I will be better equipped to understand it.
That being said, I really want to achieve the goal that I wanted to reach today. I think that I am going stay up and work until I am done.... It is feasible... I'll work on that as if there was no tomorrow. Tonight is the night.
Update:
I lost my mental focus and started to feel tired so I gave up the battle for tonight. However, I resolved the remaining mystery...
cycle 2. Stage 4, day 18:
it is not the first time this is happening where implementing a design that I have in mind takes much more time than I initially estimated.
However, I did some pretty good refactoring that puts down solid foundation for what comes next.
I am pretty satisfied of what I have accomplished in the last 2 days... I am very eager to see how well the next version will do when deployed...
23 improvements added in 2 days. The results that I got lately did boost my productivity. It is very rewarding to improve the system as small improvement can make big difference.
I was waiting for a very specific milestone before upgrading the system but I think that I am going to upgrade right away. The initial upgrade plan was with the assumption that it would be quick to reach which turns out to be inaccurate.
23 improvements may make a positive difference to the bottom line while I sleep or eat...
cycle 2. Stage 4, day 19:
The day is still pretty young and possibly plenty of other things worthwhile journaling about could happen but I got something important to report.
The new system version with its 23 new improvements did run the whole night.
In some new ways, it did not go so well. Upon the log analysis, the disappointing results did allow me to discover another very old bug that has been dormant for at least a year... (A year in a half in my best estimate). It did remain undiscovered because it did not have a big impact on regular size trades but for my new small trade mode, it makes the difference between profit and loss...
With some incredible luck, the system keep getting bogged down on the bug and this is how I did finally notice. The luck given by UMS is very methodical... Not everything happen all at once... If it was happening like that, it would be overwhelming... No instead, it gives one drop of luck at a time, just when it is the right time and you are going to be receptive to it...
I have stuffed 6 new improvements. Stopped the system and I have restarted it with yet another fixed release
cycle 2. Stage 4, day 20:
I do not fully realize how shitty the situation yet.
A cornerstone element of my very lucrative strategy just got pulled away by the exchange. Presumably, they have changed it due to the now very low ETH price.
I have not been fast enough to implement the idea that was supposed to multiply the profits of the miraculous strategy...
The system activity went from several hundreds of transactions per hour to dead silence. All that just by having the minimum volume of a single pair increased by 10.
I don't know how to take this drawback. If my success was relying only on a single digit, I was not succeeding. I was just lucky...
This miraculous frenzy at least did allow me to stress test my system like I have never done it before so I guess the outcome is still a positive one...
Hopefully, I'll figure it out and find a plan b...
Hey, lano1106!
I'm reading your journal with interest and I really appreciate your determination on achieving your monetary indipendente (I too am running UMSv2), but I suggest you to diversify your investments and your strategies, because if you only rely on crypto you are more suggestible to fluctuations and setbacks ?
I think that you want to make your trading system perfect before going over anything else and that's fair, but I wanted to suggest you this ?
I know that you mean well and I partly agree with what you are saying.
I disagree when you suggest that perhaps I should diversify.
My goal that I have set to myself is to achieve financial success through my passion that is crypto trading.
Setbacks are normal and expected. I have the strong belief that successful experts in their fields are the ones that have never given up to achieve mastery in their craft.
Where you have a point is my desire for perfection. You are partly right on this.
I got an improvement idea that took a full day to implement last Saterday. The new code is clearly superior to what it was before but concretely, it does zero difference to the system performance.
Monday major setback did remind me something. That opportunities aren't going to wait for me to be ready. I did not feel the urgency enough to really focus on what needs to be taken care first to grab the one that was in front of me. Fortunately, I have a second chance and I intend to not make the same mistake twice.
OTOH, some fields in software requires a very high standard of quality very close to perfection. Like software running medical equipment, aircrafts and managing money. If you close your eyes on small glitches, you let the door open to bigger issues... and I don't want that in a software managing my money. I wouldn't feel at ease leaving my software run unattended to go to sleep or going out if I did not have the strong belief that it is perfect AFAIK.
cycle 2. Stage 4, day 22:
The edit order feature that I was working on Monday when the sky has fallen has been completed.
It has been tested with the little activity that remains available. it seems to be working fine and this is a feature that might help down the road.
I have put into operation the first key element of my plan B. I am excited and confident that this will be working fine. I have retrieved my confidence.
One positive effect of very little trading, it is that you are not worried to interrupt the system to upgrade and restart it.
cycle 2. Stage 4, day 26:
I have been slowed down by the new features breaking the system in new ways since last journal entry.
I have mixed feelings about the current situation. Each time that my system breaks due to all the right circumstances that appear to be more than just a coincidence, I am grateful to life showing me where I made errors and oversights on important details.
I guess that it is expected when you introduce new big features to experience a period of instability at first so that you can address all the glitches.... OTOH I am growing impatient... IDK what to think about the current situation... I feel like teased and cheated by life by having made me tasted success for a very short period and then suddenly when I was starting to win at the game, the rules got changed...
Life is a constant struggle... It won't be easy...
cycle 2. Stage 4, day 27:
I have more or less analysed execution logs and hunt for new glitches for the whole day.
I am tired of being slowed down by glitches but this is a necessary evil. Yes, I am annoyed by all these glitches because I have the impression to play some sort of whack-a-mole game...
OTOH, I am grateful for the luck that I have that the exact right conditions happens in order for me to find those glitches... This is close to a miracle...
I have fixed 3 glitches... One of them was very nasty but silent. It did stop the system to consider a substantial chunk of possible trades at each iteration. This problem has been present for a very long time. Possibly more than a year...
So who knows, maybe when I am going to restart the system with the fixes in, I may end up being blown away by the fix...
On the other fronts, stuff starts to stabilize... My new partitioning code has successfully being used in a trade that did complete... (but there were few glitches in the execution... This was the initial reason to look into the logs before I start stumbling into more issues....)
So I guess that despite not being where I dream I could be, I am at least in the right direction...
Stage 4 is hard on me in this cycle... Hopefully the second month of stage 4 will be more enjoyable...
cycle 2. Stage 4, day 28:
I have tried to complete all the changes that I wanted in the new release until very late yesterday evening. I was very tired and when I have installed the new server and it started trading. It was totally broken. I was so tired that I couldn't think straight so I have decided to pull the plug and leave the server down for the night and look back at the problem first thing in the morning.
I think that when I went to bed, in the first 2 minutes, i knew exactly where the problem was and when I woke up, my intuition was right and I have fixed it in less than 5 minutes.
Now, the nasty silent bug that could have influenced the server performance that I did mention in last entry. I have looked back in the change logs. It has been introduced on 2021-10-15. I would be curious to look what program I was running back then and what I was thinking around that date but without being close to the performance of the fee free trading era... I would say that having this problem gone, it makes my system trades more than with the bug.
I have to be cautious about this conclusion... I have been bitten so many times in the past where I have introduced new features that was happening exactly at the same time with market conditions that was the real cause of the system increased activity.
It will take few days before I am 100% sure about this but so far, it looks good...
Update:
on 2021-10-15, I was running UMSv2. Stage 2 of the first cycle. I have been running UMSv2 for much longer than I was realizing...
It is funny to read what I was doing back then. I was working on a complex feature that I thought back then would change my world. In reality, it is rarely triggered...
Dreaming, thinking big and never doubt that eventually success will happen, I guess that it is the right mindset to have to never give up... Eventually, it will work...
Update 2:
It seems like having taken care of all the urgent fires and that I can return to develop the new feature that could be a game-changer, I feel like some mild procrastination is starting to kick in... Proof of that, I have been writing here instead of working on my system...
I'll kick myself to shake me out of that state... I want to work fast on completing the next feature to see what it does...
Update 3:
I took most of the day resolving another mysterious glitch. I am really obsessed about squashing them all... This annoys me but it is a really important task to get a successful and reliable algorithmic trading system...
cycle 2. Stage 4, day 29:
We are midday and I have spent half of the day so far to polish the fixes made yesterday.
sometimes changing stuff around can break things. This is what did happen and this did allow me clean some old code and improve its clarity to avoid similar breakage in the future.
I am taking a small break to lunch and the rest of the day will be dedicated to move forward my new game changing feature. Hopefully, I could have something to try out by the end of the day.
Concerning the impression that I did unlock something nice by fixing the nasty bug yesterday. That was just that an impression. It seems like I a special skill to make the intro of perceived important new code changes exactly when markets are overexcited.
Increased system activity appears to be a precursor to a market dump which is what did happen today with the BTC dropping below $20,000 again...
cycle 2. Stage 4 (month 2), day 1:
My market making subsystem is halfway done. It has 2 parts. 1. It is the detection module where its job is to detect opportunities. The other part is the execution part where the subsystem take action to benefit from the detected opportunity.
So the detection part is completed. I have a strong conviction that the idea is viable but I have been wrong so often that I want to validate by just running the detection part. If it detects nothing, it will save me the effort of creating the execution part. If it is a detection frenzy, it will motivate me to complete ASAP the missing part.
I was about to test the new code when I got interrupted by my system crashing. It did stumble into new bugs. On one hand, I guess that it is part of a normal dev cycle. You add new feature, this is going to stretch the limits of how the code is used and it will inevitably experience situations that did never happen in the past. The discovered bugs were good ones but that being said, I am growing impatient... I wish that I did not have to handle all those bugs that pretty much occupy at least half of my time... This is ridiculously slowing me down....
On the bright side, I prefer fix those issues ASAP when my system activity is low. It makes log analysis much easier...
cycle 2. Stage 4 (month 2), day 2:
I did some progress toward the completion of my market making feature. However during midday, I got again distracted by a bug....
What a drain of time. This one took me some time to figure out... About 3-4 hours... No idea why I did not spot it before. It must have been around for at least a year... I needed to have the right conditions to see it. It did happen today...
Hopefully, I won't stumble on other bugs until I am done...
OTOH, I should not be unhappy when bug fixes are served on a silver plate.... It is a blessing...