A main goal of this thesis was to design and identify an appropriate mathematical model of stochastic biological system. A CTMC model show its ability to provide a well description of biological system and its behaviour. We have demonstrated that how designed CTMC is used to describe two biological case studies: LV and repressilator. In repressilator model, the simulated data produced from CTMC model is consistent with original observed data and show similar behaviour. The main contribution of this thesis made on the part of Bayesian inference method. The problem of intractable likelihood prohibit the direct Bayesian inference, alternatively, ABC and PMMH were employed as an approximated methods. A variety of an existing algorithms were reviewed and critically discussed. The most challenging problem can be faced when employing the Bayesian inference using the proposed approaches is how to design and tuning algorithm to achieve an efficient estimation, also to balance an accuracy and computational cost. There are some aspects of the model specification that mitigate as much as possible thenumber of modes in the likelihood The effective of proposed scheme demonstrated on two biological systems. The designing ABC algorithm based on rejection and generalised rejection technique proof it ability to infer uncertainties in LV model. Furthermore, ABC algorithm within sequential version demonstrated on LV and provide a better approximation to the model parameter posterior. In order to assess the performance of ABC algorithm, a PMMH is considered as the second inference, yet, designing algorithm for PMMH more complex and difficult than ABC and require a special tuning. A resulting parameter inference of LV model obtained from employing ABC and PMMH algorithms given an experimental data that generated from artificially CTMC model are compared. An application of those proposed approaches also demonstrated on more complex case study repressilator compared to the simple LV case study. In our application, we illustrated the important algorithmic parameter that can influence the efficiency of performance of algorithm. In attempt to enable a best performance of PMMH algorithm, a nontrivial potential practical issues with its possible solution were investigated. This particle issues associated with number of particles, mixing and stationary distribution are mitigated by monitoring the convergence of several indecent Markov chains and tuning a proposal distribution to be adaptive which adjust it through the iteration to have a shape that similar to to the current posterior approximation. An ABC SMC was applied on repressilator model given both simulated and observed data, to obtain more efficient approximation to the parameter posterior, a particle diversity maintained by monitoring the the ESS. The ABC SMC suffer from the exact problem, where the exact approximation obtained if the tolerance is equal to zero which in practice impossible. To obtain a sufficient approximation we aim to obtain a smallest tolerance value which often require a very large number of simulation. In attempt to save computational time, we carried out a pilot runs to draw an idea about the possible low tolerance can be achieved. This can able us to stop algorithm. Despite the stochastic nature of repressilator model, we notice that different independent runs of the ABC SMC algorithm with exact algorithmic setting lead to similar parameter approximation. Hence, we demonstrate that ABC SMC algorithm converge to its stationary distributeion. The ABC SMC method seems to be more appropriate inference method which offer a reasonable trade-off between accuracy of estimation and computational time. In addition, we have demonstrated the applicability of the ABC SMC scheme to different cases with both simulate and observed dataset without requiring a special tuning.