lano1106 BASE v2.1 journal

[lano1106]

I got pissed at the gym this afternoon. Since the reopening, we have to reserve our spot in a class on the gym website. It is required because before CoVid, it was openbar and you were guaranteed a spot. Now they are limited to 10.

So I did reserved my spot for the 5'O clock class. Upon arriving, the coach says that I'm not registered... I vigorously contest because I did register and I had my spot. At first, I suspect a glitch in the reservation system. How could he miss a name in a list of 10 people? It did make feel bad. I took the blame. Maybe I did register to some other class some other day because I did the registration with not too much attention. He said to not worry he will manage to find a way to make things work with 11 persons.

After 10 minutes in, I have a small break, I have the time to check my phone to see if I have the registration email. I do. I go see the coach to show him. And he says that he did recheck the names and I was right to say that I was registered but we are still 11 in the class with 10 spots.

What I think that did happen was that a buddy of the coach was in standby to take a spot in the full class if someone registered didn't show up.

But what did really pissed me off is when he did ask me to take a 8 feet square floor spot delimited with tape at one end of the gym and to use the pull-up bar at the very opposite side of the gym because that is all that is available.

I contested again this BS. When your performance is evaluated on time taken, this setup was clearly penalizing me and my spot was reserved. No idea why I ended up being the one penalized. If he would have apologized for the situation and acknowledged it kinda sucked for me, I would have probably let it go but instead, he did tell rudely: This is how it is. Deal with it...

I'm mad about that. I think that I will complain to the owner about that...

[lano1106]

Stage 6 day 22:

I didn't put back my server online yesterday. Too much work to do before doing that.

My final client test went flawlessly. Therefore the testing strategy did prove to be very valuable.

I'm thinking about implementing the same strategy to test my execution engine. So far, I'm facing 2 mysteries. All I have been able to do for them was to improve the details in the logs with the hope of having a better insight into the issue next time it happens but if I was able to playback at will the events that have led to those inexplicable behavior, I am very confident that I would solve those mysteries in a flash.

I really think that I'm going to work on implementing this idea. I'm not too sure, however, what would be the priority of that task.

My guts tells me to do it right away and this is very probable that this is what I'm going to do but it isn't 100% crystal clear that this is the right thing to do.

Another anomaly that I have found is that I ended up with a negative balance for one of the currency that I trade (ETH). It is the second time that it happens. Theoretically, this isn't supposed to happen. If you send an order to the exchange, they valid the request and it is supposed to send you back an 'Insufficient funds' error if that reflects your account balance reality. Therefore, you should never be allowed to execute an order if the result is a negative balance.

I know exactly why it happens. They don't use the same number precision at the order validation then the one they use to debit your account. For debiting your account, they use less decimals and round the value to their favor... I don't like this way of doing things. It almost feels like cheating. They should get paid with the order fee and nothing else. period.

I have an email from their support laying in my inbox. I need to check if it doesn't contains explanations about the negative balance thing.

In summary, I did validate the good working of the client. I did finish analyzing the latest server run and I did identify at least one small glitch in the server that I'm going to fix next.

All is good, we keep moving forward!

Oh wow... Only 10 days left to BASE... It is less than 2 weeks... I better change gear and work faster. I wanted to have something working before the end of the program...

[lano1106]

Stage 6 day 23:

I did relaunch the server yesterday.

The client did froze this morning with something that has been working fine for at least 5 months.

I have been sidetracked to investigate this new BS problem. It feels like some sort of whack-a-mole game. I guess this is to be expected. The bigger the project is, the more probable that some hidden issues pop out without warning.

No new trading opportunity did show up so far...

[lano1106]

Stage 6, day 24:

Yesterday new client issue got resolved. TBH, it was something that has never been tested before. Therefore this is why it wasn't working.

This morning, there was a lot volatility in the crypto markets. BTC price went below $9000 USD. This has trigger a trade. Something went wrong. The problem made me found a small race condition that went unnoticed so far in the execution engine.

Those 2 problems are fixed. This morning, my system is better than yesterday. Pretty soon, it will work like a charm without any issue for several days in a row...

[lano1106]

Stage 6, day 25:

I did fix more things in the client yesterday. I did notice that the timestamps included in the exchange are now in the future. I did wrote some code to get logs proving this suspicion and it did turn out to be correct. The timestamps are about 250 ms in the future.

Considering the connection RTT and that the local clock is perfectly synchronized with NTP, this is obviously wrong.

I wrote to the exchange support about my discovery and I have an answer in my mailbox.

but beside that, I got a small moment of despair. Something did change in the market conditions that has significantly diminish the developed strategy potential.

To summarize, the first time that my system did start to trade was early March. That must correspond to BASE stage 3. Since then, I never stopped to discover glitches and improve the trade execution. That was super motivating because many trades were triggered daily. With capping trade size to about $100, there was enough trading to put my last 30 days trade volume up to $10,000 USD or more.

I did let the server operation go for about 3 weeks in June to focus on developing new client features and now that I have resumed server operation, trades have become very rare. I got 2-3 last week-end (and one of them has been triggered by fake data sent out during the exchange maintenance window) and I had 1 yesterday.

The potential has dried up during June... Once I have made this realization, it kinda remove the motivation to make the system work and improve if what it has been designed to do has stopped working. This morning, I did lower the trade trigger bar to see if it change something. Or else, I'll need to steer in a different direction fast.

That is where I am this morning.

Update: It has been confirmed by the exchange support staff that they had an issue with their book updates timestamps. They are off by about 200-300 ms.

[lano1106]

Stage 6, day 28:

Since last entry, I did not move much the project. Friday, it was a family day at the beach. Saturday, there was a birthday in the family that pretty much took the whole day off. during the day and during the birthday dinner, I drank much more alcohol than usual. This could explain why yesterday I was feeling tired and energyless during the whole day. I did sneeze a lot. Put those 2 symptoms together, it is either the alcohol and strong allergies mixed together or I caught a cold from my social interactions Saturday.

Nonetheless. I fix 2 glitches in the client. I need to wait for some trading occurring to see if the fixes did work. I did a very minor but easy to fix bug in the SQL code.

Beyond those not that important tasks. I did tackle 2 major issues.

1. I need insight about why trading has dried up so much. It is either market conditions, more competitions. A bigger player paying smaller trading fees than me having a similar strategy will always take the opportunities before they even appear on my radar. One last option to explain this problem, something has changed in my code. Actually, a very important factor did change. I have few MAs to provide some rough trend indicators, I did gradually increase their periods over the weeks so that they give me better indication inside the time horizon that I'm working in. Since trends play a role in the trade opportunity detection, changing those parameters may be behind what I'm seeing today.

I have added some logs in my code to have a better understanding of this issue. Bottom line, I'll probably need to increase the taken risks to have a chance to play.

For example, I have installed those logs in the server just before starting to write this entry. As I write, I keep a window open where I can monitor the logs, the best detected opportunity so far is in negative territory. My plan value calculation assume taker fees which is a bit inaccurate. During execution, with a favorable trend, I'll get the lower maker fee. I'll need to account for those for a more accurate plan value calculation. But even then, I may need to enter into a plan that is currently in negative territory by betting that during execution with trends moving in the right directions, it becomes positive. That is the type of refinement that my system will probably need...

2. Finally, I have undertaken the task to refactor the execution engine. This is a major and delicate task (maybe this explains why I have postponed doing it until now!) but so many possibilities will be unlocked by completing this. For one, it will improve its maintainability by breaking it into smaller building blocks. This will allow me to unit test parts of the engine to finally fix the 2 elusive bugs that I have seen happening for which I have no clue what are causing them and also, I have features in my todo list that are dependent on the engine refactoring.

Update: It seems like the Heisenberg uncertainty principle is going on in my investigation. As soon as I started to monitor the detected but rejected trading opportunities. 2 nice ones did show up since...

[lano1106]

Stage 6, day 30:

Things are going fine... I'm almost done so I think it is the time for program impressions.

I didn't bring my project that I started 6 months ago to completion but it doesn't matter. The 2 big takeaways that I get from having gone through BASE are this:

1. My debt did melt despite everything. Lockdown, minimum focus to actually make money
2. My problem solving skill went to overdrive. With it, my ability to transform a vision, thoughts and ideas into reality. It is the power of the will.

And based on those realizations, BASE was worth every penny that I did pay for it...

I wanted write that down ASAP to not forget about it. Now lets see what I can squeeze and realize into those 2 remaining program days...

[lano1106]

Stage 6, day 31:

Interesting event today. There was a trade initiated this morning. Not all operations were 100% smooth, one order didn't execute in one shot and the next step was therefore broken down in 2 parts. Not equal parts. One was using the minimum trade volume. The problem when you end up buying the minimum volume is that you cannot increase the bid price as with the available fund would force you to buy less than the minimum volume.

The end result of that is an open order that is on deep in the book since this morning for about 15h so far.

That is actually a good thing. I have a open trade dialog box on the client side and some feature in it requires open trades to test it. I did iron out the remaining bugs in that dialog box.

[lano1106]

Stage 6, day 32:

This was actually yesterday but I worked so hard yesterday that I didn't have the time to write the entry.

Normally, I would just skip the day and wait for the next time where I have time and interesting things to say but because this is the last entry of the journal (except if I do additional runs), it was kinda important to not skip it.

The reason why I was so busy is because I had this very long task. Not hard to accomplish but numbingly boring. It did require a lot of determination to go through. Today, Im joining friends for the day and the night in shack located in the forest. I wanted to have peace of mind with this taks done.

Mission accomplished!

Really, I don't have more to say about my overall program impressions than what I said in day #30 entry. (2 posts ago). That was a very good run and really the appropriate program to use for the job that I wanted done. It is not over yet but BASE did follow me well along this incredible journey of building a new source of income. The state of the project that I have been able to reach in 6 months is impressive and quite satisfying.

As of today, I started OF. I have 3 loops in so far. I started the playback as I went to bed (yeah not a full night. Some sort of 3 hours long big nap) and I have already something to write about. So next thing I do is to create my new OF journal