Speaking and listening are natural language skills which the brain is has evolved to use. As you've accurately pointed out, all babies/infants begin with speaking and listening and often don't learn to read or write until they start school. Reading and writing are artificial language skills, which is why they must be taught (either by a teacher or self-instructed) and cannot simply be naturally picked up like speaking and listening can.

I strongly advise against using Romaji if you actually want to learn Japanese. It may seem easy at first, but it causes other problems, including:
  • Greater risk of phonemic interference. This is always a tell-tale sign as to which of my students have bothered to learn Hiragana and which ones haven't. Those who are still relying on Romaji are far more likely to transfer English phonemes over to Japanese. e.g. mispronouncing まえ (mae) as "may" instead of "mah-eh" -- students are misreading the "ae" as in "maelstrom." The fact is that the Japanese /ae/ vowel set does not exist in English phonology, and also, Japanese generally doesn't slur vowels as English does. Learning via Hiragana helps you to better master basic pronunciation better than Romaji does.

  • It is often difficult to wean off dependency on Romanisation and become competent in reading/writing Japanese script. I once met a guy who'd been living in Japan for over a decade but was utterly illiterate in Japanese because he'd learnt using Romaji and couldn't wean himself off it. He could speak the language fluently, but was useless at reading/writing - so even for simple things like reading signs, menus etc. -- forget it.

Imagine if a Japanese person tried learning English using purely Katakana-ised イングリッシュ (Ingurisshu).