The human brain, which is considered to be the prototypal biological computer, in its current incarnation is the result of more than a billion years of evolution. Its main functions have always been to regulate the internal milieu and to help the organism/being to survive and reproduce. With growing complexity, the brain has adopted a number of design principles that serve to maximize its efficiency in performing a broad range of tasks. The physical computer, on the other hand, has had only 200 years or so to evolve, and its perceived purpose is considerably different and far more constrained - that is, to solve a set of mathematical functions. This picture is rapidly changing however. One may argue that the functions of brains and computers are converging. This transition comes at a critical time when the roadmap for physical computing is becoming murky after a long period of exponential growth. Hence the existential questions arise if the underlying design principles may converge or cross-breed, or if the different mechanisms (physics versus biology) will always translate into radically different solutions. Neuromorphic systems are just one possible form of cross-fertilization between biological and physical computing. Other neural concepts at different levels of abstraction can help inspire us to rethink how to efficiently perform a number of meaningful tasks and functions. This leads to the topic of this monograph. This monograph reviews some of the insights arising from both computational neuroscience and computer engineering, and evaluates how these could combine to help us build a next generation of computing systems. To create insights and identify opportunities, this monograph firstly puts neural and physical computing face-to-face, and compares how they arose, how they differ right now with respect to a number of metrics such as computational and power density, and how these metrics may change over the future decades. A similar analysis is performed at the architectural/computational model level. While doing so, ground truths in terms of obtainable performance, bandwidth and power/energy efficiency are established. Thereafter, a number of neural design principles that may translate into design guidelines for future computers are identified. On close examination of these and other observations, it becomes apparent that cross-fertilization between the domains is already happening at multiple levels, albeit in an incremental way. The publication completes with perspectives on where brain-inspired computing may lead us, some speculative bets, and a number of forward-looking reflections.