The spread of artificial intelligence (Al) over diverse industries provides many benefits as well as challenges. The inner working of an Al system still behaves like a black-box, and its adoption depends on converting it to a more glass-box structure. Recent developments in solar photovoltaic (PV) power generation forecasting indicate that Al has great potential for predicting solar power output. Interpretation of a PV power generation forecasting will enhance the efficiency and the adoption of PV energy further. This paper presents the use case of PV energy forecasting utilizing an explainable AI (XAI) tool on a high-resolution dataset. The forecasting of power generation is done using the XGBoost algorithm, and feature contributions are explained with the ELI5 XAI tooL XGBoost and ELI5 together provide simple, fast, and efficient forecasting to facilitate straightforward deployment. The proposed models are trained and tested using all features, as well as a subset of features. The results of these two models are evaluated in terms of root mean squared error (RMSE) scores.