From my point of view, one of the most difficult aspects of learning a new language is the speaking part, especially for those that are not so widespread. The main reason is that speaking needs a lot of practice and someone to talk to and in many cases, that person is unavailable. English seems to be the easiest to learn, because you can hear it everywhere: in music, news, books, tv shows, movies and so on. However, there are others which you are not in contact with so much. A good example is German. I started to learn it when I was eight years old and I learned this language for eleven years. I was very good at reading and also at writing tests, but I had some issues with speaking.
Some people can consider the writing part to be almost as difficult as the speaking one. Both of them imply knowing grammar and being familiar with wide range of vocabulary. Reading a lot of books and articles can help you develop coherence, understand grammar better and learn a lot of new expressions. It also teaches you to express your ideas clearly and corectly. which is important for a good impresion. When compared to speaking, which requiers a promt response, writing takes longer and thuse gives you the oportunity to think things over, order your ideas and express them in a better way. For example, I had to read a lot of books and articles from German magazines to get used to the vocabulary, but in the end I was able to produse good essays for the exams that I passed.
In conclusion, I think that those skills are the hardest because they imply the most difficult aspects of learning a new language: grammar and vocabulary and it takes a lot of time to master them.