What you're feeling is normal.
I resonate with your feelings in a non-ApoE4 context - as a person who is 39 years old, without children, every time one of my younger brothers or a 25-year-old cousin of mine announces their pregnancy. At those times, I feel a weird mix of happiness for them, sadness from wanting children but not having them, and guilty feelings from not being completely overjoyed for these people I love.
At those times, I talk to someone I trust, and then focus on what I can do - for example, in my case, I'm researching methylation and methyl folate therapy for my homozygous MTHFR mutation (and how it relates to childbearing).
You have done well by finding out about your genetics, and doing something about it! And my friend reminded me (today in fact, when I told her about my MTHFR mutation), we all have mutations, so if it's not one, it's another. ApoE4 is one you can do something about!