It's completely legal to invest your IRA in rental property or other types of real estate. It's also completely legal for the brokerage or bank managing your account to confine your investment options to mutual funds. Usually if you want to include real estate in your IRA portfolio, you have to set up a self-directed IRA.
A self-directed traditional or Roth IRA allows you to invest in anything you want, with exceptions such as collectibles. As the name implies, you research and select investments without any guidance from the account manager. If you want your IRA to buy rental property, you have to find the property you want to buy and evaluate what it's worth. Then you direct the manager to make the purchase, using IRA funds. Legally, your IRA, not you, is the owner.
Buying and Owning
If, say, you start your self-directed IRA this year with a $5,500 contribution, that's all you have to work with in buying real estate. You can't use your non-IRA funds to help make a purchase, and you can't get a conventional mortgage loan. Once you buy the rental property, your IRA has to pay for all taxes, repairs and other fees. Investing in multi-tenant properties is often a good choice. That way if one tenant goes, you have others providing an income stream to keep your IRA in the black.
IRA investments aren't magic. If you buy real estate through an IRA, use the same due diligence and thorough research you do with other investments. Fraud is rife in the self-directed IRA world, so be suspicious of anyone who assures you an investment can't lose money, or claims an abnormally high rate of return. It's often a good idea to keep a reserve fund inside your IRA in case you have an unexpected emergency at the rental property.
When you own a rental property, you can use it yourself or offer it to family at a cut rate. That's off the table when your self-directed IRA is the owner. The IRS has a flat ban on self-dealing, meaning that you and your IRA cannot do business together -- for example, you cannot sell your house to the IRA, then pay the IRA rent to keep living there. If you're caught self-dealing, the IRS will dissolve the IRA. Everything in it instantly becomes taxable income for the year.