Cycle 4, stage 3, day 10:
Yesterday, I have started thinking about designing a feature that would power cycle the trading session with the exchange.
The idea is good as it adds another level of safety net in case something goes wrong. It would theoritically allow the system to return in a sane state once problems are detected.
OTOH, I had a small voice telling me that it was not worth the effort to implement this for a once every 5 years occurence. Finally, it did strike me that my idea was addressing the symptom and not the cause and I have finally realized what did cause the mess that the system did experience.
It was coming from a core component. A component that is among the very first that I wrote when I started the project. It works fine when there are no errors but for some input, it can corrupt the system state.
Fixing this has been a very unpleasant experience because it is quite a complex element having interactions with several other components and because it has been a while since I created it, I had rediscover how it works... I have documented its design this time to make revisiting this component easier the next time the need arise.
I am now pretty much done with this task. I believe that now the problem is gone and this whole episode did give me the opportunity to improve that component...
This whole task did take the whole day and I have not done anything else... but I think that this is a really good thing to have work on and tomorrow, I'll be in a better position knowing that I can move forward on a more solid foundation.
Cycle 4, stage 3, day 14:
since last entry things went smoothly... I have deployed the new system release successfully. I was a bit concerned that I could have broken something during the refactoring but everything went nicely. While everything was on autopilot, I have started to work on migrating my system on server closer to the new exchange location...
this task did offer few unique challenge... I have improved my instructions on how configure a new machine to host the system. This has forced me to refresh my mind... I am going to move my system from Seattle to LA area... As an intermediary step and because I wanted to try out the migration process, I have a bunch of servers left and right and I had 2 serves in San Jose... Upgrading my first server in San Jose to be able to host the system has been challenging... I did few tweaks to the setup as I went along that I did not consider important enough to document only to eventually forget about them and being bitten by their absence on the target server...
I had 1 last config snag and it required remote hands to fix it. It should be done by now but I'll probably will not restart my system on the new server but I did run my test latency client from the new machine and it is very exciting. round time trip despite been at mid distance between my Seattle location and the ultimate location in the LA area did give me the best RTT that I have never seen before... It could be very fun to run... and it makes me hope that with a machine in LA, I could become again competitive with the big traders having the privilege to bypass cloudflare.
While I was working on the migration, after several profitable trades, my system did encounter 2 new problems... Both were 2 small 1 liner bugs... Those type of bugs are fun to fix... This makes me very optimist for the future because for those small bugs to show up, it was requiring a good amount of actions which what my system did experience. The next version containing those fixes is going to be even more robust.
Totally unrelated but I am using the Brave browser and I have a couple of websites registered with Brave as a publisher... I am really not sure how the publisher side works but all in all, this was giving some small ridiculous of BAT every months. 0.25 BAT or so... but I think that yesterday, I did receive something like 10 BAT!
I am not sure if it is good idea coming from UMS or if it is some sort of procrastination but I got a new strategy idea... I have written down the big lines of it in a design document to not forget about it. From here, there is 2 possible paths. I can either write a small proof of concept prototype to validate if the idea is valid and has potential or I can leave it there and continue support of exchange #2 that I have been talking about for months now without never dedicate myself 100% to the completion of that effort...
It is hard to decide... I know that sooner or later, it is going to be crucial to diversify my operations... It is a risk to be on a single exchange... No one can predict who is going to be the next FTX...
OTOH, the new idea if valid could generate $300-$1000 income daily or more and that finally eliminate all my financial concerns instantly...
I am in a good pass... It seems like my effort and decision are starting to pay off... As if I was in the process to put in place the last few pieces of jigsaw puzzle...
and finally on a more personal note, after having spent pretty much the last 3 months like a dedicated monk, I have started feel the urge to maybe to get a new female friend... This need did never occur during that period... I was way too much concerned about succeeding which was more or a survival matter and also since I have pretty much hit the bottom of the barrel financially, I kinda was feeling unworthy to have a FWB... I guess that now that I am starting to see the light at the end of the tunnel... my confidence is resurfacing... I got a match maybe 2 weeks ago on a dating app... First impression was positive and I was thinking that something would come out of this meeting... Things went sour and ended up without ever meeting in real life... I think that I have avoided a drama queen so not a big loss... but still this has teased me even more that having a FWB would be a nice thing to have... nothing to report until yesterday morning... I have matched with very interesting, very hot, very naughty women in the same half hour...
I am amazed how the simple fact of wanting something will give an experience similar to crossing a desert but the moment you decide to get rid of that desire, abundance is offered to you on a silver plate... I am not committed to any of them but tbh, considering the small amount of time that I want to dedicate to social and fun... exploring 3 leads is more than I wanted... I'll try to make my mind fast with who I am more interested in... but bottomline, the result is so spectucular and out of nowhere... I wonder if I am not experiencing some TID from another program... not impossible since I am starting to approach the end of my 4th UMS cycle...
Cycle 4, stage 3, day 16:
I sent a cease and desist letter to a company that was billing me for a service that in my opinion was not delivered few months ago...
They sent me an offer to settle this issue by offering a refund of $500...
If they offer me this right away without going for fight, that must be because I must have a strong case... I was claiming them $10,000...
but I will accept the offer. The whole point of my letter was to make them stop to harass me with invoices for a service that they were not doing and were unable to prove otherwise. If they leave me alone, I can say mission accomplish...
Getting a small amount on top of it, this a bonus! A small amount of money that was totally unexpected and is coming my way...
Cycle 4, stage 3, day 21:
Since the last entry, I did a lot of changes to fix problems discovered during last week run...
It appears that those changes did break the system... Yesterday, in front of this observation, I was feeling a bit down...
I am in the process of completing cleaning the mess that I have accidentally introduced... As very often when this type of situation occurs, this is an opportunity to discover unsuspected new ways to improve the code that I would not have discovered without searching the code to fix the current bug...
which is pretty cool...
Cycle 4, stage 3, day 24:
Progress has been slow in the last few days. I did not feel a lot of energy. If I put that aside, I had a brilliant refactoring idea to get rid of an annoying small problem that was making my client not respect the exchange request rate and as a consequence was putting my client on the penalty bench once in a while. It was not causing big problems now but it could have if I was forbidden to trade because of the issue while some situation would require fast reaction...
I did complete this refactoring task this morning. It took a little bit longer than I expected...
I still have few things to fix but hopefully, I'll be able to try it out sometime tonight...
Beside that, I did experience a lot luck. When I now consult the log file, it has become so monstrously big (50+ MB) that unless I go for something specific in it, stumbling into something interesting inadvertently is essentially like finding a needle in a haystack. In the last few days, I think that this has happen not 1, not 2 but 3 times that I have stumble into some very interesting bugs that went unnoticed before their discovery.
Cycle 4, stage 3, day 25:
Yesterday, I have stumbled into an issue that I could not make any sense out of it. I have slept on it and this morning, things got clearer and I have fixed the issue.
Cycle 4, stage 3, day 27:
Yesterday evening, an unscheduled exchange server maintenance session did mess with my client.
This did allow me to uncover and correct 6 new bugs but I spent the whole day doing this which kinda annoy me...
A small change that I did introduce over the week-end appears to be a beneficial addition...
Now, I hope that problems and issues leave me alone for few days so I can focus on the profitable tasks...
Cycle 4, stage 3, day 31:
Last day of stage 3...
Things have moved significantly in the last month...
1 take away from the last month is that my latest strategy which was looking very promising 1 month ago... despite adding a lot of polishing, it seems like the more I was working on perfecting the strategy, the less it was returning profit...
As of today, this is a mystery... It is either that market conditions did change and are making the markets a totally different beast to what they were a month ago...
Or, I did break something down the road without realizing it...
I will spend some time to look into this weird experience... That would be so cool, that I realize that I broke something, fix it and finally get decent results....
If I put that aside, my project is progressing... I am just not too sure that I am taking the shortest path possible to achieve the goal...
I remember back in January... I had this 1 feature that I wanted to add that was possibly a gamechanger feature... This intent got sidetracked by a newer shinier idea on which I did work since then...
This is either listening my subconscious mind and be open to opportunities or else this is some sort of procrastination where I find anything else to do than working on the initial plan...
Technically to close the UMS loop correctly, 2 months of stage 4 still remains... I start to crave for a new program... but unless UMS goal is achieved, I cannot afford to purchase another program... So idk what I will do in 2 months... The optimistic scenario is a major breakthrough in the next 2 months which is far from being impossible...
the next 2 months are going to be interesting...
Cycle 4, stage 4, day 2:
there is a major refactoring that I have done last month. I must have worked on it for maybe a good week on it. This was to enable a cool feature that would allow my system to trade for many accounts simultaneously. This is also opening the option of assigning specific trading strategies to specific accounts...
I was about to start using this new feature when I have discovered an omission. A data structure that should also be refactored to support this new feature. Discovering this omission has been like opening a pandora box. By working on addressing the omission made me find other issues that could not be triggered before the feature is actually used...
I am almost done but fixing this has been tedious and painful because it is affecting a big module that is a core component. It is possibly comparable to a full day open heart surgery in terms of complexity and by how demanding in terms of mental effort...
I have spent the last 3 days on this... I am exhausted from the effort... Next task will be to figure out why my latest strategy was producing a lot more trades 1 month and half month ago when it was only a protoype than the more refined version that is running today...
I will review daily reports for the last month or 2 to try to discover a pattern... I will also reread this journal... I recall that I had the thought that the first few days of each month did appear to be more quiet than usual... What I am seeing now appears to corroborate that observation but this is not the only cause for sure... Hopefully, I'll find a better explanation...
Cycle 4, stage 4, day 4:
Today, I did what I said that I would do that is performing a meticulous review of past previous daily reports to try makes sense of my latest strategy performance drop in the last month... I believe that this is the first time that I am doing such an extensive analysis on past daily trade results...
I am starting to get some insights into the situation and perhaps that I will have a solution for it... Definitely my best month was February (Stage 2). I forgot how good it was but it was good. The strategy was generating about 5x more profitable trades than what it is making currently.
My server did core dump on me this evening. The core dump reason was original. My thread 64KB stack got depleted. It is the first time it happens in 3 years. I try to avoid using the heap and I stuff as much as I can on the stack... Analyzing the core dump made me discover that a library function was wasting 3Kb of stack space. I did fix the guilty function by refactoring it and this is going to give an extra 3KB to the stack of all the threads going through this library.
I have also increased the thread stack size to 96KB to be extra safe. This extra small stack size is an old habit I got when I was working on a server launching hundreds of threads... The stack size was back then a concern for memory usage... I do not have this restraint anymore but I kept this habit and it kinda force me to keep stuff slick...
Cycle 4, stage 4, day 5:
The day is still young so it is not impossible that I have more to journal later but I woke up this morning with a realization that is important enough to write it down as I have this thought fresh in my mind. I have the impression that stage 4 in this 4th cycle is pushing me further by giving me inspiration/new ideas...
After 5 days in stage 4, I am feeling good with what I am feeling is coming to me financially in the near future...
I have completed my analysis. It is not 100% confirming the hypothesis that I started to have yesterday... but the idea may still have some merit to explore...
A small change that I did yesterday has apparently broken something... I'll need to write a small test program to validate but I think that I am facing a stupid unsigned/signed integer conversion...
Cycle 4, stage 4, day 7:
today is the second day without any electricity at my place. I am writing down this entry offline to post it later when electricity comes back eventually.
Wednesday evening a lot of icy rain did cause a major electric blackout where I live. Needless to say that this is a major show stopper when it come to work with my computer. No electricity, no Internet, no phone.
My main worry is that the blackout last more than 48h. If it does, I am afraid that I may lose all my frozen food.
With this concern aside, the situation did force me to adapt what I do with my time. Yes, I try to visualize what I am going to do on the code once the electricity comes back. I hope that having done so will somehow make things go faster by having rehearsed them mentally first.
Most of my time in the last 2 days have been used for reading. I have so many books that it is even not funny. Something that I have noticed is that my book selection has drastically changed.
I have exclusively picked finance related books. I usually opt for programming books. When you think about it, this is perfectly logical. Yes I am enjoying reading programming books but I am already an expert. Exiting my comfort zone and improving my application domain knowledge is probably the most profitable thing that I have made this week.
but this is a blessing and a curse at the same time. My reading is giving me so much ideas on ways that I could improve my system while it is impossible to work on them. It is very frustrating.
In those situation, I realize that my backup plan in case of a major electric blackout is deficient... OTOH, does the effort to plan for something happening once every 20 years worth it??
Cycle 4, stage 4, day 8:
electricity went back yesterday evening.
First thing that I did was to open my computers and check what my trading server did while it was running unsupervised.
it did not bad... but it did crash few hours before electricity came back at my place...
I started to look at the logs how it did happen. I have not been able to find out yesterday... but this morning, I have finally ended up to figure out what did happen.
I started using the decorator design pattern on the order execution strategy object. Until yesterday, the need to undecorate a strategy did never happen and there was a bug in the undecoration function...
No idea why... I must have had a blind spot in my mind for this option until now but during my last few days reading, I have realized that my system was almost not using much market indicators...
My next task is to add the ADX indicator into my system...
During my shower, there has been a burst of trading activity. The system did make 35 trades in a 3 minutes interval. When I came back to my computer, this situation was so unusual, that my initial reaction was to freak out thinking that I inserted another bug... After taking a closer look, all was fine, the system made about $10 of profits trading LTC with a roughly $20 budget....
I wish this was like that all the time... but maybe it will soon...
Cycle 4, stage 4, day 13:
I have completed a refactoring that is a direct result of my reading last week during the electric blackout.
A minor aesthetic change was that I did rename what I was calling a strategy to 'Alpha Model' which is the name used in books and in the industry for the concept that I did label Strategy.
If I ever onboard an industry expert, using the standard labels might help.
but this change was really minor. Once I started changing name, I figured that I could further by breaking down a big blob of code into smaller sub-components. This idea is already giving back benefits. With the computation of a new market indicator, this gave me an idea for a new alpha model. This one has the potential of generating big amounts of profits... but the point is that the last refactoring is making it even easier to plug in a new model into the existing framework...
The required extra computational work that the new indicator requires is very minimal. However, the model that was generating a lot of trades in February. It has been very quiet only to restart generating a good amount of trades since last week-end... I think that the increased activity is due to the increased of market volatility and activity (BTC and ETH have crossed over $30K and $2K) but I am not 100% sure since my client timing did also change at the same time...
Yesterday, I was in the process of planning making a quick prototype of my new model idea but I got interrupted by execution glitches that needs to be analyzed... I find this a very bad timing... On one hand, my system has been incredibly stable for the last weeks so it is a bit frustrating that problems happens right when I was about to complete something very exciting. OTOH, I absolutely need to look right away those issues ASAP when the problem is fresh. Otherwise, if I postpone the investigation, I know by experience that the problems will end up not being investigated or since the system keep evolving, the logs will start to not be in sync with what is in the code...
So far, what I did fix by the log analysis were good catches. In an ideal world, analysis would be completed promptly and enough time will remain to complete my prototype today...